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ABSTRACT 

The study involved an attempt "to promote use of 
evaluation data in compensatory education through a technical 
assistance effort at the state education agencies (SEA) level. It 
included a secondary analysis of Hawaiian Title I longitudinal data 
and dissemination of findings through a series of consultations with 
evaluation and program personnel at the SEA and local education 
agencies (LEA) levels. Impetus for the study came from increased 
awareness of information needs on the part of project staff for 
program design and improvement purposes. The study suggests several 
potent variables for predicting Title I student achievement. These 
variables include school type, school/project enrollment, pretest 
achievement -status, per pupil cost, absenteeism, project setting and 
instructional approach. Outcomes of the study suggest that 
face-to-face technical assistance is not only a viable but perhaps 
one of the most effective ways of promoting use of evaluations. 
(Author/PN) 
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Promoting Evaluation Use Through 

\ 

Technical Assistance: An SEA Example 
Abstract 

The study involved an attempt to promote use of evaluation data in 
compensatory education through a technical assistance effort at the SEA 
level. It, included a secondary analysis of longitudinal data and 
dissemination of findings through a series of consultations with 
evaluation and program personnel at the SEA and LEA levels. Impetus for 
the study came from increased awareness of information needs on the part 
of project staff for program design and improvement purposes. Outcomes 
of the study suggest that face-to-face technical assistance is not only a 
viable but perhaps one of the most effective ways of promoting use of 
evaluations. 



Promoting Evaluation use Through 
Technical Assistance: An SEA Example 

INTRODUCTION 

Evaluation use is often defined as using information gathered in an 
evaluation as the basis for decision making. Under this definition 
little evidence can be found to support the assertion that evaluation 
information is being used by decision makers (Wise, 1978; Thompson and 
King, 1981) • Some researchers have made a distinction between use of 
evaluation which determines go/no go decisions from use which influences 
a policy maker's thinking about a program without putting evaluation 
information to a specific, documented use (see Rich, 1977). 

A more relaxed, and "perhaps more realistic, definition of evaluation 
use is 9f fered by Alkin et al. (1979) . According to these authors, use 
occurs when evaluation provides people with* information that influences 
them in making a decision about the future content, practice, appearance 
or existence of the program. 

The forces which lead to evaluation use are often complex <and fluid 
(Patton, 1978; Alkin, 1979) . Hansen (1981) reviewed literature on 
evaluation use and identified four factors influencing use: technical/ 
methodological, personal/ role related, communication related, and 
political/organizational. In a survey of SEA personnel in nine western 
states he found a general perception of the technical/methodological 
factor as the major obstacle to evaluation use. A traditional tendency 



to view evaluation from a restrictive, summative perspective might- have 
contributed to such perceptions. The author suggested that evaluation 

use be viewed as a continuous variable to tease out relationships among 

> 

evaluation consumers, levels of use, and emphasis for technical 
assistance. Degrees of evaluation use included; program awareness or 
public relations, influence on or consideration in decision making, and 
major basis for decision making. 

In a ddrvey of 15 Title I districts in 6 states David (1981) found 
that the primary local uses of evaluations were to meet legal 
requirements, to provide feedback, and to provide gross indicators of 
program effectiveness. Title I evaluations did not seem to serve, as 
primary purposes, either as a basis on which to judge the prograjp or as a 
guide to program improvement. Reasons for the minimal use of evaluations 
in judging program effectiveness reflected preferences for measures of 
achievement other than standardized' tests, a fear of misleading 
comparisons, and the view that programs have multiple goals. The author 
also found Title I programs to be quite statfle, thus limiting the 
potential for changes to occur, regardless of whether such changes are 
based on evaluation informatipn. 

David (1981) suggested, that merely improving the technical quality of 
measures would not by itself affect the level of evaluation use. To 
increase use one must address the underlying reasons for lack of use, 
including individual attitudes and beliefs about the program and 
evaluation. Among other things, communication and cooperation between 
program staff and evaluation staff needs to be strengthened. Results 



should be presented in person if they ere to be clearly understood and 
hence used by staff. Local district staff need assistance in 

incorporating evaluation information into planning and decision making. 

— * 

In much the same vein, vanderFloeg (1982) described increment*! 



improvement, feedback and formative evaluation as the life-blood of local 



contact and direct involvement of the Valuation staff with project 
administrators (vanderPloeg, 1982; Holley, 1980). 

In a recent national survey, Alkin et al. (1982) concluded that both 
the Title I Evaluation and Reporting System (TIERS) data and other types 
of Title I evaluation data were used at all decision levels by state 
education agencies (SEAs) and local education agencies (LEAs) . In their 
report to the U.S. Department of Education, the researchers maintained 
that the Title I evaluation system did, indeed, have utility. They 
uncovered strong evidence that evaluation data were s4en as an 
information source in the daily life cycle of Title I projects. 
Evaluation data contributed in incremental ways to major program 
decisions. At the SEA level, evaluation data were used to monitor LEA 
compliance, to recognize both problem areas and exemplary programs and to 
influence administrative and curricular actions. LEAs typically used 
Title I evaluation data to change attitudes and opinions toward Title I 
projects, to recognize situations requiring attention and to contribute 
to decisions on administrative and curricular actions. 

The researchers found that different kinds of evaluation data had 
relative utility at the various organizational levels. School boards, 
district advisory committees and external agencies relied on summative 



evaluation. Evaluation 
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data* such as TIERS data, more extensively than other evaluation data. 
At the district administrative level, TIERS data were mixed about equally 
with other Title I evaluation data developed by the! district. At the 
building level, principals, coordinators and the like relied slightly 
more on TIERS than on other data. At the classroom level, TIERS data 
were less often used. Instead, data more closely related to the 

instructional programs were preferred. 

♦ » * 

Analysis, of case studies showed that evaluation use was affected by 
several contextual variables, including; 

Evaluator credibility . The reputation and credibility of th« 
evaluator is an important determinant of use. While evaluators may 
achieve credibility in differing ways they must be perceived as 
competent and trustworthy. 

Evaluator commitment to use . Credibility, while important, is not 
enough to insure evaluation use. The evaluatpr must also have a 
commitment to seeing that evaluation results are used by decision 
•| makers. 

Interest in evaluation by decision makers and the community . 
Evaluation data are used when they are tailored to the needs and 
interests of the local school community. Use occurs when evaluators 
draw relevant information from TIERS data and when they conduct 
special evaluations to meet local requests. 

Local focus of evaluation . Use increases when evaluation^ are 
specifically designed to meet local needs. Success of use is 
attributable to timely response and sensitivity to local concerns. 
Effective presentation of results . Graphic, narrative and 
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nontechnical nodes of presentation increase the utilization of 
evaluation data to local decision aakers« 

Assistance in developing procedures for the use of evaluation data . 
Evaluation use increases when decision makers are assisted in 
understanding how they might use the evaluation data. Successful 
evfiluators typically proyide detailed, step-by-stei* procedures to 
potential users. 

Alkin et al. (1982) suggested several recommendations for improving 
evaluation utility as Title I became Chapter I in tfce new law. They 
believed that Chapter I evaluation utility could be enhanced by continued 
technical assistance in reformatting "TIERS-like" results to meet LEA 
information , needs. Also, SEA and LEA evaluation units should be 
encouraged to design a variety of local decision-focused evaluation 
strategies. In particular, locally designed evaluation procedures might 
provide information on the impact and costs of various materials and 
processes within projects. 

The researchers pointed out that many local and state agency 
personnel required guidance in developing procedures to follow when 
making decisions. It was not that administrators did not want to use 
relevant information. They typically did not know how to incorporate the 
information into their decision processes. Moreover, evaluators must 
become aware of the vital role their personal style played in evaluation 
utilisation. Training procedures for evaluators might emphasize the 
evalua tor's role and the Importance of interpersonal skills. 



OBJECTIVE 

The primary objective of the present study was to promote evaluation 
use by the Hawaii Department of Education for program planning and 
improvement purposes. Hawaii has systematically gathered Title I 
evaluation data since the inception of evaluation requirements in the 
Title I legislation more than ten years ago. Little, however, was done 
toi integrate the data either on a longitudinal or cross sectional basis 
to address questions relating to statewide program activities. It was a 
widely shared perception within the DOE that evaluation data gathered 
during the past several years had not been ( put to maximum use. There was 
« need to find out what types of data had been collected and how 
dat^ fould be used to address substantive issues relating to Hawaii's * 
overall /Title I effort. 

The stjidy was designed to address various substantive questions on 

'& 

f the basis of a secondary analysis of the evaluation data, oust Several 



years. It was hoped that the project would serve u a 



prototype model 



for evaluation use at the SEA level in a cost effect iv^ manner. Such 
secondary analyses could be made part of regular TAC services if their 
value to SEA personnel was fully demonstrated in the present project. 

The primary audience of the project was to be the Hawaii SEA 
personnel (e.g., evaluation and compensatory education staff). It became 
clear, however, the district coordinators and Title I school 
administrators as well as the teaching staff could also benefit from tjie 
project. Plans were subsequently made to disseminate the results to a 
much larger audience than the state office atal^f . 



PROCEDURE 

The project was carried out by the Technical Assistance Center (TAC) 
in cooperation with the SEA evaluation and compensatory education staff. 
The involvement of compensatory education staff was particularly crucial 
with respect jbo formulating and prioritizing questions to be addressed. 
The scope of the secondary analysis depended in a large measure on 
feedback provided by the compensatory education staff. 

The study was conceptualized in early 1981 when data use became an 
area of interest to both SEA and LEA staff in Hawaii. The SEA evaluation 
staff perceived a need to build a data base by pooling data presented in 
evaluation reports for past school years and to use the data base to 
address statewide issues relating to Title I. Several consultations were 
held during the early months of 1981 which resulted ih the delineation of 
the following design elements: 

1. Data sources would be limited to evaluation reports prepared by 
the external evaluator and related documents (e.g., state 
directories and welfare reports). 
• 2. The study would cover data for the 1978-81 school years. 

3. Schools would be used as the unit of analysis. Data were to be 
aggregated across grades for each school. (This decision was 
later relaxed a bit to accommodate grade-by-grade analysis of 
achievement data.) 
A preliminary list of variables of interest was compiled by SEA 
evaluation staff on the basis of a review of several school-level 
evaluation reports. The list was reviewed by TAC staff and subsequently 



7 lu 



served as a basis for the development of a cpding format. Both 
evaluation ahd compensatory education staff were involved in formulating 
and prioritizing research questions for the study. Following several 
discussions, it was decided that while all variables identified were of 
interest tor the staff, the study would focus on questions relating to the 
effects of various instructional approaches and project settings on 
student achievement. 

Data coding was accomplished in three phases. Preliminary cooing of 
projects and school information was conducted by TAC interns at the 
Laboratory in Portland. These data pertained essentially to information 
obtainable from the district-level reports. Such information items 
included school code, school enrollment, school type, grade level covered 
by school, welfare status of attendance areas and overall student 
achievement. g 

The second phase of data coding was conducted on site in Hawaii with 
the aid of three graduate students hired by the compensatory education 
section. The coding covered some 300 plus school-level evaluation 
reports and included information obtainable only from the school- level 
reports. Such information included student grouping, diagnostic testing, 
inservice training, project setting and instructional approach. With 
respect to the coding of instructional approaches a list of basal 
jMttrials used in Title I projects was prepared by the compensatory 
education staff. The list was reviewed by language arts specialises in 
the Office pf Instructional Services. Based on this review, two primary 
categories of instructional materials/ approaches were established i 
(a) materials which were incorporated in a prescribed^, instructional 
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system and (b) Materials which were not part of a prescribed 
instructional system. Examples under the first category included Anne 
Adams, the Chicago Mastery Program and the Hawaii English Program. 
Examples under the second category were^SRA's word attack skills kit, 
McGraw Hill's Webster skill cards and the Reader's Digest^reading 
skill-builder. 

Two major categories of instructional settings were identified: 
(a) regular classroom and (b) pull-out. In the regular classroom 
setting, certified teachers and/or aides worked with Titl| I students in 
the regular clasjroo*, providing tutoring and other instructional 



activities. In the pull-out setting, project students were pulled out 



/ 

from the classroom to receive Title I instruction in a separate roo* for. . } 



learning center, usually in small groups. The students were sent baJk *to 
the regular classroom after the Title I instruction. 

The third phase of data coding included grade-by-grade achievement 
data within each project. This was done following considerable 
discussion on comparability of NCE gains across grades, resource 
constraints and the merits of having grade-by-grade analysis of n 
achievement data. The coding was accomplished by student helpers kt the 
evaluation section under the supervision of the SEA evaluation stiff. 

As would be expected, a number of problems were encountered in coding 
information from the evaluation reports, including: 

1. Missing data * Information was not provided in the district or 

/ 

school-level reports. For example, there were no data on staff 
PTE and absenteeirti for the 1978-79 school. year. 
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2. Ambiguous narratives . Some project narratives irr the 
school-level reports were difficult to interpret.* For example , * 

* in somp cases membership of parent 'advisory cou^ils was not 
clearly described. . ,/ 

3. Lack of discreteness^ M any projects used composite project 

, • settings and student, groupings. For instance, some narratives 

•j mentioned use of individual, small group and large group 

instruction^ with various grouping configurations. 

Host problems were resolved on the basis of the data coders,' best 

judgment. In other cases, the data sheets were left blank. 

Completed data sheets were mailed, to TAG staff at the Laboratory in * 

Portland. These were keypunched and quality control measures were taken 

> 

to ensure accuracy of coding and validity of the coded data. A few 
cases were discarded because 'of excessive missing data and some 
inaccuracies in coding were corrected. Data analyses wer* then performed 
on the "clean" data. 

Initially, means and standard deviatiofc were computed for all 
variables of interest. These calculations were conducted separably for 
each of the three years included in the study ^ Descriptive statistics 
were obtained on the selected variables by subject area (!•*•# reading 
versus mathematics) and by school type (i.e., public versus private). 
Due to the relatively small number of private schools, subsequent 
analyses were confined to public school samples. 

Correlational analyses were then performed on samples of public 
schools, again separately for e£ch of the three school years. These 
analyses were conducted to identify factors which were related to student 
achievement. 



A third set of analyses consisted of analyses of covariance (ANCOVAs) 
of project NCE gains usihg per pupil cost and pretest NCE as covariates, 
the latter variables having been shown to be related to student 
achievement. The ANCOVAs were performed to assess the effects of project 
settings and instructional approaches commonly used in Title I projects 
in Hawaii. Project settings included (a) regular clasroom and (b) 
pull-out and (c) combinations. Again, these analyses were performed 

separately for each of th^ three school years. The ANCOVAs were first 

.... v . * 

performed on NCE gains for projects as a whole — grade level gains having 
been aggregated across grades. Similar analyses were^then conducted 
separately for each grade. , 

RESULTS 

Due to space limitations, only the major findings and conclusions 
will be presented here. The study suggests several potent variables for 
predicting Title I student achievement. These variables include school 
type, school/project enrollment, pretest achievement status, per pupil 

0 

cost, absenteeism, project setting and instructional approach. Results 
of the study support the following conclusions: 

1. Project impact as measured by NCE gains over the three' school 
yegxs 9ovpred by the study has remained quite stable ranging 
from 7 NGEs to 8 NCEs for the reading projects and holering 
around U NCEs for v the math projects. 

2. Stj&dents in the lower grade levels can be expected to make 
greater NCE gains than students in higher grade levels. The 
difference is probably a result of the developmental process 
rather than a consequence of differential program effectiveness. 

• ' ii 
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3. Size o| project and/or school enrollment is inversely related to 
achievement gains. This suggests that small projects located in 
small schools seem more conducive to learning basic skills than 

. large projects located in large schools. 

4. Students who scored lower on the pretest tend to make greater 
achievement gains than tfceir higher scoring counterparts. This, 
again, is probably due to developmental factors rattier than 
program effectiveness. 

5. Per pupil cost is positively related to achievement gains. 

' 'Undoubtedly, tfiis occurs because per pupil cost is a surrogate 
for prdgram elements (e.g., more experienced teaching staff, 
better facilities and materials) which tend to produce high 
achievement. 

6. Ther^ is some evidence that absenteeism is inversely related to 
achievement gains. This finding is generally in congruence with 
results of recent time-on- task studies. 

7. In terms of achievement gain, students who received title I 

» » 

instruction in the pull-out setting are likely to perform as well 

as, if not better than, their counterparts in other settings. 

This suggests that despite its apparent drawbacks (e.g., loss of 

regular classroom instruction, transition time) the pull-out 

setting remains a viable, option for providing services to Title I 

youngsters. 

8. Students who received Title I instruction under a combination of 
instructional approaches (e.g., a prescribed system supplemented 
by miscellaneous materials; are likely to make as much, if not 
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greater, achievement gains aa their counterparts under other single 
approaches. This perhaps points to the validity of allowing the teaching 
, staff flexibility in usinig materials in ways most suitable for individual 
students within a project. 

The primary areas of interest in the present study consist of such 
manipulable program features as project setting and instructional \ 
approach. Contrary to negative perceptions expressed by some researchers 
(Harnischfeger, 1980), the pull-out setting was shown to be superior to 
other project settings .(e.g., regular classroom) examined in the study. 
Even when grade level effects were taken into account, thef overall trend f 

h 

still appeared to favor the pull-out setting. The trend was less clear 
with respect to the effects of different instructional approaches. In a 
majority of instances, however, the combination approach was shown to 

have worked as well as, if not better than, other approaches. 

X 

DISSEMINATION 

Findings of the secondary analysis were presented to TAG client* in 
narrative, tabular and graphical formats. (See Appendix A for an 
example.) They were first reported to the SEA evaluation and 
compensatory education staff. Input was obtained from the SEA staff with 
respect to accuracy and raeaningf ulnjpss of the results. Minor revision^ 
were subsequently made in the final report for wider dissemination. 
Compensatory education staff in each LEA were given a copy of the written 
report as well as an ofcal briefing on the findings. In several 
districts, the results wpre disseminated in an abbreviated form (see 
Appendix A) to Chapter I teachers and school administrators. 



ERIC " tf lb 



Natural^ the effects of this technical assistance effort on 
decision-making has yet to be fully demonstrated. However, based on 
Hansen's (1981) delineation of levels of use, the awareness/public 
relations function of evaluation appeared to have been achieved. 
Moreover, based on anecdotal data, the findings have had son* influence 
on the thinking of district administrators in at; least a cOuple of cases 
where the district administration was on the verge of switching from 
pull-out to the regular\lassroom sitting for Chapter I projects. 
Reports from SEA compensatory education staff who monitor the district 
Chapter I projects further confirmed that district personnel were taking 
the findings of the study into consideration in planning Chapter I 
projects. At the SEA level, plans are being developed to expand the data 
base on an on-going basis, adding annual evaluation information to the 
data base and conducting the secondary analysis on a yearly basis, • 

SUMMARY AND CONCLUSIONS 
The study involved an attempt to promote use of evaluation data in 
compensatory education through a technical assistance effort at the SEA 
level. It included a secondary analysis of longitudinal data and 
dissemination of findings through a series of consultations with 
evaluation and program personnel at the SEA and LEA levels. Impetus for 
the study ^came from increased awareness of information needs on the part 
of project staff for program design and improvement purposes. The 
research strategy was similar to what Klausmeier (1982) described as 
improvement-oriented research. First, it was directed toward educational 
improvement through evaluation use. Second, it was planned and carried 
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out cooperatively with users of the results. The study also bore 
resemblance to what Schwab (1977) labeled as practical research that 
the objective of the research was to aid practitioners in making informed 
decisions concerning means of improving the education of students 
enrolled in Chapter I projects. 

Findings of the study, along with their implications for program 
design and improvement, were disseminated through a series of 
consultations with SEA and LEA staff in the state. Given the successful 
working relationships. between TAC and its clients, the study exemplified 
the effects of -contextual variables affecting evaluation use as described 
j?c^ Alkin et al. (1982) • Of the six contextual variables, conwitment to 
utilization and the developofent of procedures for evaluation use are 
integral parts of the TAC mission. After years of providing technical 
assistance to SEA and LEA personnel, TAC staff have established good 
rapport and working relationships with clients and achieved credibility 
as givers of sound advice. Interest in evaluation by decision makers /and 

local focus of evaluation were quite evident in the present study. 

*> 

Finally, results were presented in narrative, tabular and* graphical forms 
to achieve maximum impact on potential users of information. 
Undoubtedly, these conditions of evaluation use were enhanced by the 
face-to-face contact between T&C staff and clients. Once the conditions 
conducive to use were established, the stage was set for TAC staff and 
clients to work in concert to promote use of evaluation information which 
would ultimately benefit youngsters enrolled in Chapter I projects. 
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HAHAII CONDUCTS SECONDARY ANALYSIS 



Th. Hawaii Stat. Dapartiawt of Eduction r.c.ntly compl.tad • a.condary 
•nlyai. of Ch.pt. r 1 data gatharad ov.r • thr—y.ar p. r iod. Growing 
out of an lncraaaad .wa»«naa. of inform.tion naada for a*Xing program 
d~i.ion. at th. .chool L.1, th. .tudy addra.a«j a.v.r.1 quaationa, including 

1. What ar. th. .ff.cta of diffarwt project aatting. on .tud.nt 
achiavwiMt? 

2. . What art th. of foot, of difforont instructional approachaa * 

on student achisvwant? 
Data -r. obtainad fro. valuation raport. for ..cm. 300 Chaptar 1 projact. 
implamantrt during tha 1978-81 .chool yaar. in th.' di.trict.. Th. 

—jor projact aattinga conaiatad of pull-out, ragular claaaroom and aoma 
combination of tha two. Inatructionaj approach., include praacrib* ay.tama, 
non-pra«»ibad ayatama and aom. combination of th. two. 



».ault» of th. an^y.i. augga.t that .tudant. racaiTin, Chaptar X inatruction 
in th. pull-out ..tting ar. lUciy to perform a. wall Mf if Mt ^tfr tnan, 
thair counfrp^tt. in oth«r matting.. Itudanf racking inatruction und.r 
a combination of approachaa (a.g., a praacribad ayatam aupplamant* by 
■imcallaaaoua matarial.) ara likaly to achi.r. v.11 a,, lf ^ imtt ^ x 
than, thair count^parta undar othur approachaa. 



Th. raaulta ar. diaplayad graphically in tha figuraa balow 
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NCE Gain /o* Reading Projects^ 
By Project Setting 




1978-79 



1979-80 



1980-81 



I y regular classroom 
pull- out 
combination 
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1979-80 



1980 - 81 



NCE Gain for Reading Projects 
By Instructional Approach 



sXXX 



CZ3 Instructional materials only 
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